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EXECUTIVE SUMMARY 

The Partnership for Assessment of Readiness for College and Careers (PARCC) commissioned a 
multi-part study to determine the viability of using the drawing response interaction on the 
PARCC Mathematics Assessment. This study in particular focused on students with disabilities. 


PARCC has over 40 interaction types on the summative assessments. Why introduce a drawing 
response interaction? There are several drivers to the addition of this functionality. The first is 
comparability. While scores across modes are comparable overall, they could be stronger at the 
lower grades. Students who respond to constructed response on paper can provide drawings. 
Feedback from students in lower grades from the mode comparability study indicated the 
desire for a drawing tool. Data from the scoring of paper responses indicates up to 10% of the 
responses include a drawing at grade 3. The Common Core State Standards (CCSS) and National 
Council of Teachers of Mathematics (NCTM) both indicate that students should have multiple 
solution paths and tools available to them. 


A usability study was conducted by the Pearson’s Research and Innovation Network (RIN) and 
Assessment Solutions and Design departments to assess the usability of a drawing response 
interaction type with general education and special education students with disabilities. 
Thirteen participants in grades 4 through 6 (during the 2015-2016 school year) participated in 
this study; seven students had a disability (low vision or fine motor impairment). Participants 
completed a series of math questions on a Chromebook or iPad that included the drawing 
response interaction, while observation and video recording took place. 


The goal of this study was to test the usability of the drawing response interaction feature and 
determine if and how it assists participants in problem solving. Specifically, this study sought to 
compare observations from students without disabilities to students with fine motor 
impairments or low vision. 


Observations/Findings Based on the Drawing Response Interaction: 


e Participants from all groups (fine motor impairment, low vision, and general population) 
could generally use the drawing response interaction to complete basic tasks such as 
drawing a straight line. 

e Participants from the fine motor impairment group and general population group were 
generally able to locate and discern between the drawing response interaction box and 
the equation editor textbox. 

e Participants from all groups were able to erase their work using one or more of the 
erasing methods. 
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e Participants experienced difficulty identifying and using the “x” eraser tool and straight 
line tool. 

e Participants were able to use touch and/or the mouse more effectively than the 
trackpad for drawing. 

e The differences in drawings on the drawing response interaction and paper/pencil were 
likely not enough to impact scoring. The fine motor impairment group produced slightly 
more readable drawings using paper/pencil, and the low vision group produced slightly 
more readable drawings using the iPad. 


General Observations: 


e Participants from the fine motor impairment group were very challenged by the math 
content and, at times, were unable to complete tasks because of this. 

e Participants from the low vision group struggled significantly with zooming to read and 
respond to the tasks. They had difficulty determining where they could and could not 
zoom. 

e Participants from the fine motor impairment group and low vision group completed 
drawing tasks very slowly. They finished much less of the test than the general 
population group. 


Recommendations: 


e@ Move forward with Component 3 of Study 4. 

e Evaluate making adjustments and enhancements such as increasing the point size of the 
drawing response interaction box border and make the straight line/pencil tools more 
distinguishable; adjust the “x” eraser tool to enable draggable erasing; and Include a 
library of shapes (rectangle, square, circle, triangle) and other tools to assist in the 
efficiency of using the drawing response interaction. 

e Include in the Teacher Administration Manual that we recommend the use of 
touchscreen or mouse for students using the drawing response interaction, particularly 
for younger students. 

e Consider adding guidance in Appendix M of the PARCC Accessibility Features and 
Accommodations manual around screen size minimums and browser/operating system 
combinations for low vision students, particularly those using 3x or 4x browser zoom. A 
study could be used to inform the guidance. 


INTRODUCTION 


The PARCC Tests assess participants’ current performance on rigorous state standards for math 
and Language Arts; the tests are designed to not only assess for correctness, but to also look for 
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evidence that a participant understands and can apply their knowledge. The goal of the study 
was to gather feedback on a drawing response interaction from general education and special 
education students, which may be integrated into the PARCC assessment. Pearson is working 
with PARCC to deliver a flexible drawing response interaction feature to enable participants to 
record and submit their handwritten work and answers for math problems. 


Purpose 

The focus of this study was to investigate the usability and accessibility of new item interactions 
and functionalities on different devices and for various student groups. In particular, we 
wanted to determine the usability of the tool among participants who have a fine motor 
impairment or low vision, as compared to participants who do not have a disability. 


The team recorded observations about physical interactions with the iPad and Chromebook, 
with a focus on the usability of the drawing response interaction. 


Test Objectives 

e Do participants use the drawing response interaction to support their answer? 

e@ Do participants with fine motor impairments or low vision experience any usability, 
access, or processing issues with the drawing response interaction? 

e@ Can participants identify and use the drawing response interaction features? Without 
moderator intervention? 

e@ Do participants produce comparable results using the drawing response interaction and 
paper/pencil? 


General Observation Objectives 
e@ Do participants encounter any difficulties with the drawing response interaction? 
Which features cause issues? 
e Do participants encounter any general technological difficulties with the iPad or 
Chromebook? With the touchpad or mouse? 
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User Profile 

There were thirteen total participants, including five participants with fine motor impairments, 
two participants with low vision, and six participants without disabilities. Five of the 
participants were in fourth grade, seven were in fifth grade, and one was in sixth grade. 
Participants self-reported various levels of comfort with math. 


Participant | Grade | Gender | Disability Use of Technology Math Comfort 
computer/tablet for Used for Level (1=not 
math? Test/Input confident at all, 

Method 5=extremely 
confident) 

P1 5 Female Fine Motor Yes--DreamBox Chromebook, | 1 

Impairment (game) Touchpad 
P2 4 Female Fine Motor No Chromebook, | 3 
Impairment Touchpad 

P3 5 Female Low Vision Yes--Math Facts on iPad, Touch 3 
iPad 

P4 6 Female Low Vision No iPad, Touch 3 

PS 4 Female Fine Motor NA Chromebook, | NA 

Impairment Touchpad 
P6 5 Female Fine Motor Yes— Math Facts iPad, Touch 5 (Note: She may 
Impairment not have 


understood the 
question, as she 


also said she 
struggled with 
math) 
P7 5 Male Fine Motor No iPad, Touch 3 
Impairment 
P8 4 Male None Yes--Games iPad, Touch 4 
P9 4 Male None Yes—Math Facts iPad, Touch 4 
P10 4 Male None Yes--First in Math Chromebook, | 4 
Mouse and 
iPad, Touch 
P11 5 Male None Yes--Calculator iPad, Touch 3 
P12 5 Male None Yes—First in Math, iPad, Touch 4 


PARCC 
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P13 5 Male None No iPad, Touch 5 


METHODS 


The usability test was completed within 60 minutes. The test was designed to investigate the 
usability and accessibility of the drawing response interaction on different devices and for 
various student groups. Each participant session was organized in the same way to facilitate 
consistency. Three participants used the Chromebook and ten participants used the iPad. 
Video and audio recordings and moderator/note-taker observations were used to collect data. 


Recruitment 

Pearson worked closely with Trinell Bowman to identify districts to target. Anne Arundel 
County Public Schools and the Maryland School for the Blind were selected and primary points 
of contact were identified. We worked with Dr. Joshua Irzek to recruit low vision participants 
from the Maryland School for the Blind. We worked with Ms. Leigh Mann to recruit general 
education students, students with fine motor impairments, and students with low vision from 
Anne Arundel County Public Schools, although no students with low vision agreed to 
participate. 


We sent Dr. Irzek and Leigh a recruiter screener letter (Appendix A) to explain who we were 
looking for to participate and how schools and participants could get involved. Dr. Irzek and 
Leigh recruited the participants and gathered parental consent forms (Appendix B). 


We were able to recruit six students with fine motor impairments (five participated, as one was 
ill on the testing day), two students with low vision, and six general population students. 
Because color blindness is not reported in student files, we were not able to identify or recruit 
any color blind participants. We originally wanted students who had taken PARCC on a 
computer in the past, but only one participant (P12) met this criteria. We also wanted students 
who were generally on grade level; a few of the participants from the fine motor impairment 
group seemed to be below grade level. 


Test Details 


Introduction/Demographic Interview (2 - 5 minutes) 
The moderator explained that the purpose of the test was to review online tools 
embedded in a high-stakes assessment. Participants answered questions about their 
past use of technology products in their math courses and their comfort with math. 
These questions allowed us to gain an understanding of the participant’s previous use of 
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technology with math while giving the participant a chance to get comfortable. Last, the 
moderator emphasized that we are not testing the participant’s ability to solve the math 
problems but rather their perceptions of the online tools. 


Prototype Testing / Follow Up Interview (35 - 55 minutes) 
The moderator explained that the participant should imagine that he/she is about to 
take a test online and asked them to pretend this was a high stakes test similar to one 
they would take at the end of the year. The moderator instructed the participant to 
“think aloud” as he/she completed the tasks to let us know what he/she was thinking. 
The test was delivered via Chromebook for four of the participants (P1, P2, PS and P10) 
and via iPad for ten of the participants (note: one participant, P10, used both a 
Chromebook and an iPad). 


Farewell (1 - 2 minutes) 
When the task list was finished, participants were prompted to provide any other 
relevant feedback. Participants received a $25 Walmart gift card honorarium at the end 
of the session. 


After test data had been collected, notes and video data were reviewed and coded to 
determine themes. 


Test Environment 

Usability testing was conducted during the week of November 16, 2015 at schools in the 
greater Baltimore area: Belvedere Elementary School, the Maryland School for the Blind, 
Central Elementary School and Shady Side Elementary School. Testing took place in classrooms 
or conference rooms, each equipped with a table and chairs for the moderator, note-taker, 
participant and school/district representative. For each session, the audio and screen 
movements were recorded using Camtasia. 


Test Moderator/Observer Role 
Before the session, a school representative or the moderator retrieved the participant from 
his/her classroom. 


The moderator introduced and administered the usability test, referencing a test script as 
needed. The moderator introduced the participants to the assessment and informed them of 
the new tool. The moderator explained that we were more interested in the participant’s 
perception of the tools than of their ability to answer the math questions correctly. The 
moderator explained the “think aloud” procedure for the session and answered any preliminary 
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questions. The moderator administered the test and asked the participants to rate the tasks 
completed during the session on a scale of 1 to 7, 1 being very difficult and 7 being very easy. 
At the end of the session, the moderator presented the participant with a $25 Walmart gift 
card. After the session, a school representative or the moderator walked the participant back to 
his/her classroom. 


During each interview, the moderator operated the camera and recording software. The note- 
taker took notes on the feedback from the sessions. After each interview, the moderator 
organized and posted video footage for review. 


Test Format/Prototype Overview 
The test was divided into three phases. 


1. Inthe first phase, all participants interacted with three tasks which required them to 
draw and shade basic shapes. These tasks were written specifically for this usability 
study and were designed to test whether or not participants could interact with the 
drawing response interaction on a basic level without instruction or a tutorial. 

2. In the second phase, participants interacted with a series of tasks that required them to 
solve (or attempt to solve) PARCC-like questions using the drawing response interaction. 
These questions were based on either PARCC assessment questions in which some 
students included a drawing on the paper assessment or PARCC practice and sample 
test questions. Tasks were drawn from grade 3 items. 

3. In the third phase, participants interacted with a series of tasks that required them to 
solve (or attempt to solve) a PARCC-like question using paper and pencil. These tasks 
were parallel to the items in the second phase and were included to compare 
paper/pencil responses to responses on the drawing response interaction. 


Evaluation Measures 
For each usability study objective, the following measures were used: 


e@ Observation of users’ interactions with the drawing response interaction 
e Verbal dialogue, including feedback to follow-up questions 


RESULTS 


Thirteen participants participated in PARCC’s drawing response interaction usability study. 
Video and audio recordings are available for twelve of the thirteen interviews (see Appendix C). 
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Due to motor impairments, one participant was only able to complete a very small portion of 
the test; we included the responses and feedback she was able to provide. 


Results are broken down by general observations and group observations (fine motor 
impairment, low vision, and general population). 


Previous Technology Usage with Mathematics 
Prior to beginning the usability test, participants were probed regarding their use of technology 
with math. 


e 4of 13 participants reported no use of technology in math 

e 8 of 13 participants reported using some sort of technology in their math courses 
@ 6 participants used technology for math games or math facts 
e 1 participant used technology for PARCC 
e 1 participant used technology for calculations 

@ 1of 13 participants did not report her technology usage 


General Perceptions of the Drawing Response Interaction 

Below is an overview of general observations from the study for all three groups (fine motor 
impairment, low vision, general population). Group-specific observations can be found in the 
Population-Specific Drawing Response Interaction Observations section below. 


General Observations 


Do participants successfully use the drawing response interaction to support their answers? 
e Ingeneral, participants could use the drawing response interaction for simple tasks. 
© Participants using the Chromebook (P1, P2, PS, P10) experienced more 
difficulty than participants using the iPad, regardless of impairment. 
12 of 13 participants were able to draw a line using the drawing response 
interaction. 
m P5 (fine motor impairment group) was unable to draw a line using a 


trackpad. 

e Participants in the fine motor impairment group (P1, P2, P5, P6, P7) struggled with the 
content of the questions. With question modification, these participations were 
generally able to use the drawing response interaction. 

Participants in the low vision group and general population group generally 
understood the content and were able to use the drawing response interaction. 
Do participants encounter any difficulties with the drawing response interaction features? 
Which features are most difficult? 
e@ Pencil/Straight Line Tool 


PARCC Drawing Response Interaction Usability Test 
11 


oO All participants correctly identified the pencil tool. 
O 2 participants correctly identified the straight line tool (P4, P13). 
© 6 participants did not correctly identify the straight line tool (P7, P3, P8, P9, 
P11, P12). 
m 4 participants thought it was an eraser (P8, P9, P11, P12). 
m P6 thought someone else controlled the computer switching between 
the pencil and straight line tool. 
e “x” Eraser Tool 
Oo 8 participants recognized the “x” tool as an eraser tool but they struggled to 
get it to work correctly and a few participants were unsuccessful and perceived 
the tool to be broken (P1, P7, P8, P9, P10, P11, P12, P13). 
m “| thought it would be an eraser but | can’t figure it out.” (P8) 
O 1 participants was not sure what the tool does (P2). 
Do participants encounter any general technological difficulties with touch or mouse? 
@ Chromebook 
Participants struggled to use the Chromebook (trackpad and/or mouse) to 
complete the tasks. 
Participants struggled to draw shapes they were satisfied with. 


Participants were generally able to use the iPad to draw shapes. 
Some participants encountered minor issues with unintentional zooming upon 
double clicking an icon. 


Do participants run out of room? 
e@ The fine motor impairment and low vision groups tended to run out of room more 
frequently than the general population group. 


Do participants locate the response area for drawing and text components? 
e@ Fine Motor Impairment Group 
oO PS had difficulty locating and staying within the drawing response interaction 
box. 
e Low Vision Group 
oO P3 and P4had difficulty seeing the borders of the drawing response interaction 
box and equation editor textbox. 
@ General Population Group 
oO. All participants were able to locate the drawing response interaction box and 
equation editor textbox. 


Do participants produce comparable results using the drawing response interaction and 
paper/pencil? 
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For all groups, the differences in drawings on the drawing response interaction and 
paper/pencil were likely not enough to impact scoring. See Appendix D for side-by- 
side comparisons of drawing response interaction and paper/pencil drawings for each 


participant. 
Fine Motor Impairment Group 
oO 5of5 participants preferred paper/pencil for basic tasks (P1, P2, P5, P6, P7). 
O Paper responses were generally slightly easier to read. 
Low Vision Group 
© Both participants preferred the iPad (P3, P4). 
oO iPad responses generally were easier to read. 
General Population Group 
3 of 6 participants preferred the iPad (P8, P9, P11). 
2 of 6 participants preferred paper/pencil (P10, P13). 
1 of 6 participants said it depended on the ask (P12). 
Paper and iPad responses were comparable. 


Task-Specific Observations 
e Basic Tasks (Tasks 1-3) 


© O participants used the straight line tool for Task 1. 
oO 1 participant used the straight line tool for Task 2 (P4). 
oO Aparticipants increased the width of the pencil for shading in Task 3 (P2, P3, 
P8, P9). 
@ Computer PARCC-like Tasks (Tasks 4-9) 
© Most participants struggled with the tools in the equation editor textbox. In 
general, they had more difficulty identifying and navigating the equation editor 
tools than the drawing response interaction tools. 
e@ Paper/Pencil PARCC-like Tasks (Tasks 10-12) 
oO Readability of responses was generally comparable on paper/pencil compared 
to the drawing response interaction. 
oO Participants showed their computational work on paper/pencil but did not 
generally do so on the drawing response interaction or in the equation editor 
textbox. 


Population-Specific Drawing Response Interaction Results 


Fine Motor Impairment Group 


There were five participants with fine motor impairments (P1, P2, P5, P6, P7). Three of the 
participants used a Chromebook/trackpad (P1, P2, P5) while two of the participants used an 
iPad (P6, P7). One participant (P5) was only able to attempt one task, as it was extremely 
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difficult for her to use the trackpad; she had very limited verbal skills, so she was unable to 
share verbal feedback with us. One participant (P2) became a bit frustrated and chose to end 
the session early. 


Tasks Attempted 


Tasks 1-3 
e 4of5 participants (P1, P2, P6, P7) completed these tasks. 
e 30of5 participants (P1, P6, P7) did not know what a curved line was. 
e@ 1 participant (P2) did not know what a rectangle was. 
Tasks 4-12 
e Participants struggled greatly with math content. The moderator modified the 
questions for most of the participants. 


Paper/Pencil vs iPad/Chromebook Preference 


Do participants prefer to draw on the iPad/Chromebook or on paper? Which is easier? 
@ Tasks 1-3: 
Oo 5of5 participants preferred paper/pencil (P1, P2, P5, P6, P7). 
m “Because on the screen the kids have less space to draw, on the 
paper/pencil, they have more space to draw.” (P1) 
m “You don’t have to press any buttons...and [the paper] has a lot of 
space.” (P6) 
e@ Task 11: 
O 1 participant preferred the iPad (P7). 
m “It’s easier to change the color.” (P7) 


Which looks better (i.e. which was easier to read/interpret)? 
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The differences in the drawings on the drawing response interaction and paper/pencil 
were likely not enough to impact scoring. 
For the Chromebook participants (P1, P2, PS), the paper/pencil responses were 
slightly more readable than the Chromebook responses. 
e@ P5, who was unable to draw responses on the Chromebook using the trackpad, 
was able to draw responses on paper. 
P1 and P2 expressed frustration with using the trackpad to draw on the 
Chromebook. 


For the iPad participants (P6, P7), the paper/pencil responses were relatively 
comparable to the iPad responses. 


Participants who used the trackpad struggled more with navigation than participations who 
used touch. 


Are participants able to sufficiently navigate the drop-down features? Click the correct 
button? 
@ Overall, participants were able to navigate the drop-down features and select the 
correct button. 
One participant (P5) experienced difficulty selecting the correct button. She had the 
most severe motor impairments. 


Are participants able to successfully complete the click, drag, release sequence? 
e Participants were able to complete the sequence on the iPad. 
e Participants struggled more with the sequence on the Chromebook/trackpad. 
oO P5 was unable to complete the sequence. 
oO Pi and P2 struggled at times with the sequence. 


Do participants run out of room? 
e 4of5 participants ran out of room or expressed that they want more room (P1, P2, 
PS, P7). 
oO Participants ran out of space on Task 1 (P5), Task 2 (P1), Task 3 (P1), Task 4 
(P1), Task 6 (P2, P7), and Task 9 (P7). 
Oo Task 2 is difficult “because the square (drawing response interaction box) is so 
small.” (P1) 
o “| think they need to make the box bigger so you'll be able to fit all of the 
tables and the people.” (Task 4, P1) 
@ 1of5 participants experienced difficulty remaining in the drawing response 
interaction box when she was drawing her responses (P5). 
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Do participants experience difficulty with shading? 


@ Overall, participants using the iPad were able to shade effectively, while participants 
using the Chromebook experienced more difficulty with shading. 


Low Vision Group 


There were two participants with low vision (P3, P4). Both participants used an iPad. Both 
participants used the browser zoom feature at its maximum magnification and held the iPad 
very close to their faces to read and respond. P3 inverted the colors on the screen. 


Tasks Attempted 


mE PPPP PP PPP Pe 
eo tT TT 
CN Co COC 


Tasks 1-3 

e Both participants were able to complete these tasks on paper and on the iPad. 
Tasks 4-12 

e Participants generally understood the content. 


Paper/Pencil vs Chromebook/iPad Preference 


Do participants prefer to draw on the Chromebook/iPad or on paper? Which is easier? 
@ Tasks 1-3: 
o Both participants preferred the iPad (P3, P4) 
m “| think it’s easier to draw a straight line on the iPad...for the pencil, 
you can’t really see what you’re drawing. Well, | can’t really see what 
I’m drawing.” (P3) 
| would use [the iPad] because you can push the line and move it 
around and make it larger.” 
@ Task 11: 
oO P4 preferred paper. 
m “| just drew instead of having to zoom in and all that stuff.” (P4) 


Which looks better (i.e. which was easier to read/interpret)? 
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e The differences in the drawings on the drawing response interaction and paper/pencil 
were likely not enough to impact scoring. 
e@ The iPad responses were slightly more readable than the paper/pencil responses. 


Are participants able to navigate adequately? 
e Zooming 
© Both participants struggled with zooming. 

m P3 and P4 unintentionally drew in the drawing response interaction box 
when they were intending to zoom or scroll. 

m P3 and P4 thought they could zoom anywhere on the screen, including 
from within the drawing response interaction box and equation editor 
box. 

© Both participants disliked having to zoom. 

m “| don’t really like having to zoom. Like when I’m reading this to get to 
the other side, | have to zoom, then let go, then zoom.” (P4) 

mg “| had to zoom a lot of times. On the test, it would take time to do 
this.” (P4) 

e Navigating to the drawing response interaction box 
© Both participants had difficulty seeing the borders of the drawing response 
interaction box. 
e Identifying the drawing response interaction box and equation editor textbox 
© Both participants were able to zoom to find the drawing response interaction 
box and equation editor textbox. 


Are participants able to find the drop-down buttons? 
e Both participants were able to find the drop-down buttons. 
e P3 could not see the difference between the pencil and straight line tools. 


Do participants run out of room? 
e@ P3 ran out of room on Tasks 2 and 3. 


General Population Group 


There were six general population participants without fine motor impairments or low vision. 
One participant (P10) used the Chromebook/mouse for a few tasks and the iPad for the rest of 
the tasks, while the rest of the participants used the iPad. 


Tasks Attempted 
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Tasks 1-3 
e All participants were able to complete these tasks on paper and on the iPad. 
e The participant using the Chromebook/mouse (P10) struggled more than the 
participants using the iPad. 
Tasks 4-12 
e Participants generally understood the content and were able to use the tools. 


Paper/Pencil vs Computer Preference 


Do participants prefer to draw on the computer/iPad/Chromebook or on paper? Which is 
easier? 
e@ Tasks 1-3: 
Oo 4of6 participants preferred the iPad (P8, P9, P11, P12). 
m “Il can make a perfectly straight line, and it takes less time to erase.” 
(P8) 
“It’s easier to shade in than to do it with a pencil, cause with a pencil, 
you can’t make the brush thicker.” (P8) 
“| think the iPad would be easier because if you want to draw straight 
and curved lines with different colors, you’d have to get different 
colored pencils and it would take longer.” (P11) 
oO 2of6 participants preferred paper/pencil (P10, P13). 
m “Paper [is] easier cause the mouse can sometimes swivel faster than 
normal.” (P10) 
e NOTE: P10 was using a Chromebook/mouse for this question 
oO 1of6 participants said it depended on the task (P12). 
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mw “...it depends, making a straight line, it would be probably be [the 
iPad], but a curved line is easier on paper and pencil.” (P12) 
e@ Tasks 5-12: 
Oo Task 11 
m “Paper, because I’m used to having paper, and | usually have more 
space on paper than on a tablet or computer.” (P10) 


Which looks better (i.e. which was easier to read/interpret)? 
e The differences in the drawings on the drawing response interaction and paper/pencil 
were likely not enough to impact scoring. 
e The paper/pencil and iPad responses were generally comparable. 


Are participants able to navigate adequately? 
All participants navigated adequately. 
All participants identified the drawing response interaction box and the equation 
editor textbox. 
All participants were able to click the buttons and complete the click, drag, release 
sequence. 


Do participants run out of room? 
@ P9 ran out of room on Task 4. 
e P10 ran out of room on Task 8. 


FINDINGS & RECOMMENDATIONS 

This section discusses positive results and potential issues that became apparent during the 
study and provides actionable design recommendations to increase the usability of the drawing 
response interaction. 


Overall, most participants were comfortable interacting with the drawing response interaction 
for simple tasks. The fine motor impairment group and low vision groups faced more 
challenges than the general population group. 


Summary of Findings 


Overall, all groups were generally able to complete basic tasks using the drawing response 


interaction. Participants were able to use touch more effectively than the trackpad for 
drawing. 
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The fine motor impairment group was generally able to navigate the drop-down features and 
select buttons. All but one participant was able to complete the click, drag, and release 
sequence. All but one of the participants ran out of room on one or more tasks. Many of the 
participants struggled with the math content. The participants using the iPad were able to 
use the drawing response interaction with more ease than the participants using the 
Chromebook/touchpad. 


The low vision group experienced the most difficulty with zooming. They struggled to identify 
where they could and could not zoom, and they were frustrated by the amount of time it 
took to zoom. Zooming negatively impacted the number of tasks they were able to complete. 
They struggled to see the edges of the drawing response interaction box. Their drawings 
were more easily readable on the iPad. 


The general population group was able to navigate adequately, was able to identify the 
drawing response interaction box and textbox, and was able to easily click the buttons and 
complete the click, drag, and release drawing sequence. On average, they were able to 
complete more tasks than the fine motor impairment and low vision groups. 


Overall Recommendations 


e@ Move forward with Component 3 of Study 4. 

e Evaluate making adjustments and enhancements such as: increasing the point size of 
the drawing response interaction box border, making the straight line/pencil tools 
more distinguishable, adjusting the “x” eraser tool to enable draggable erasing, and 
including a library of shapes (rectangle, square, circle, triangle) and other tools to 
assist in the efficiency of using the drawing response interaction. 

Include in the Teacher Administration Manual that we recommend the use of 
touchscreen or mouse for students using the drawing response interaction. 

Consider adding guidance in Appendix M of the PARCC Accessibility Features and 
Accommodations manual around screen size minimums and browser/operating 
system combinations for low vision students, particularly those using 3x or 4x browser 
zoom. A study could be used to inform the guidance. 


Participant Feedback 


The following are suggestions provided by our participants. These suggestions are based off 
of verbal feedback shared by the participants during the testing sessions. 


Drawing Response Interaction Features 
e Pencil/Straight Line Tool 
oO Create more distinction between the pencil and straight line tool icons. 
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© P4 suggested adding color to the tool icons to help distinguish between the 
two. 
e “X” Eraser Tool 
o Create a draggable eraser functionality. 
oO "| would make it so that if you drag it over the line, instead of just tapping the 
line." (P8) 
Other Feature Considerations 
e@ Shapes 
© Consider adding a library of shapes or a tool to increase the efficiency of using 
the drawing response interaction. 
oO P13 wanted a “shape fixer” that would make the shapes he drew more 
accurate. 
e Paint Bucket 
© Consider adding a “paint bucket” tool to help students shade shapes. 
o “ls there a way there’s a paint bucket that could fill [the shape] in? Select the 
color, press the corner, and it would fill in.” (P11) 


Navigation and Space 
e Location/navigation of response areas 
© Create thicker borders around the response areas to enable low vision 
students to better locate these areas. 
e@ Space 
Oo Create larger response areas to enable students with fine motor impairments 
to more easily and fully show their work. 
e Buttons 
© Increase the size of the buttons to enable all students to select the correct 
buttons. 
© Consider adding colors to the buttons for easier identification. 


Low Vision Modifications 
Enable font-size increase so students do not have to zoom. 
Enable text-to-speech. 
Increase the default thickness of the pencil to enable students to better see what they 
are drawing. 
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APPENDIX A - LETTER OF RECRUITMENT 


Dear Local Accountability Coordinator: 


Pearson, on the behalf of PARCC, Inc. is pleased to invite students who are in Grades 4 and 5 to participate in a 
research study designed to inform the future development of test questions for the Partnership for Assessment of 
Readiness for College and Careers (PARCC) consortium. 


We conduct Usability Testing with people; students, teachers, administrators, and parents. One-on-One usability 
testing with an experienced researcher allows us to get a clear understanding of how accessible, usable, and useful 
our products are. Student confidentiality is maintained at all times and their identities will not be shared with 
anyone. 


The focus of this study is to investigate new item interactions and functionalities’ usability for various student 
groups on different devices and accessible by students with special needs. The following research questions will be 
addressed through the use of PARCC-like mathematics questions: 


Does the drawing response interaction allow students to more fully and authentically demonstrate their 
knowledge and skills of mathematics and differentiate between those students with and without the 
knowledge/skills being measured? 


Is the drawing response interaction intuitive? Usable? 
Is the drawing functionality (palette) sufficient? Is there a need for instructions/tutorial? 


What are the usability, access, or processing issues for special populations, in particular, students with fine 
motor impairments, low vision and color blindness? 


We Need Students: 


We are looking for students who are functioning at or near grade-level in mathematics and who took the PARCC 
assessment online. We will be selecting a mix of students who have one or more of the following documented 
characteristics, along with a few students who have none of these characteristics: 


Low Vision — Students who used 2x-5x magnification (200% or more) on the recent PARCC online assessments. 
Colorblind or color spectrum shifts 
Fine motor impairments, who can use a mouse or paper 


Each session will take approximately 60 minutes and student participants will be offered a $25 Walmart gift card 
for their participation. All students who participate will require consent from their parent or guardian and/or 
approval by the school district. Parents and teachers are welcome to ask any questions in advance of the study. 
Someone from the school or central office should be designated to observe the sessions in person. 


Where: Various Elementary Schools 


When: November 9-20, 2015 
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Materials: Pearson will be providing a laptop or tablet for the students to use and a camera for recording the 
session. If there is other hardware students need, please let us know. We will require a room with internet 
connectivity, and preferably a phone as well. 


There will be three to five, 60-minute sessions each day. If any of those days will not work, please let us know. We 
will likely be moving from building to building each day or two, depending on how many students are available at a 
particular location. 


How to Participate: 


Please provide the following information using the attached Excel document to Rebecca Gilchrist at 
rebecca.gilchrist@pearson.com by Thursday Oct 15, 2015. For every three students with disabilities you 
recommend, please include one student who does not have a disability in one of the three areas of focus. If your 
students meet the requirements for our study, we will contact the coordinator to schedule each student's session. 


What, if any, customized settings or equipment did the student use (e.g., mouse controls, alternative mice or 
keyboards)? 


What, if any, assistive technology did the student use? 


Any other test accommodations not previously mentioned (e.g., scribe, large screen, extended time, text to 
speech) 
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Colorblind (y/n) 
If Yes, then describe (e.g., red/green, color spectrum) 


Low Vision (Y/N) 


If yes, then what magnification level (e.g., 3x) and any color perception or light sensitivity considerations? 


Fine Motor Impairment (Y/N) 


If yes, please describe student’s ability to draw simple shapes and to write by hand. 


Other Impairments (Y/N) Please list if yes (e.g., learning disability). 
Is the student classified as English Language Learner (ELL)? (Y/N) If yes, what level? 


Thank you very much for your interest in our study. We look forward to working with you and your students. If you 
have questions, please contact Rebecca Gilchrist via email at rebecca.gilchrist@pearson.com. 
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APPENDIX B — PARENTAL CONSENT FORM 


Pearson 

3075 W Ray Rd, Suite 200 
Mail Stop 210 

Chandler, AZ 85226 

Fax: 480.457.7408 


www.PearsonSchool.com 
Re: PDRE PARCC Drawing Tool Usability Testing 
Dear Parent or Guardian of Student Participant, 


In an effort to improve our educational products, Pearson Education (“Pearson”) is committed to obtaining 
feedback from students who use or might use our products. The more input users or potential users provide during 
the design process, the more likely our products will meet our users’ needs. 


Pearson is seeking participation and feedback from your child on educational products as part of a Research Study 
called the PDRE PARCC Drawing Tool Usability Testing. For this study, your child will be asked to use an iPad or 
computer application to perform predetermined tasks and to answer online survey questions about the product 
and participate in interviews. The Research Study session will last approximately 60 minutes. 


Your child’s participation in the Pearson PDRE Equation Editor Cog Labs/Usability Testing Study is optional, and 
you are under no obligation to have your child participate. However, should you agree to have your child 
participate, this letter sets forth the expectations and activities required by the participants in this Research 
Study, and sets forth the terms and conditions applicable to these activities: 


1. The study is being conducted in the state of Maryland. Within each state, locations will vary and will be 
communicated with the parent or guardian prior to the study. Locations will be in public buildings and may 
include, but not be limited to: 


° Public schools 
° Schools for the blind 


e Learning Disabilities Association Facilities 
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e Universities 
° Pearson Offices, where available 


2. As part of this Research Study, your child’s first and last names will be collected in order to organize 
and differentiate data between research participants. However, this information will be only used for that 
purpose and Pearson will destroy this information upon completion of the Study. 


3. During your child’s participation in the Research Study, your child will not encounter any more risk than 
is normally associated with using an interactive online learning application. Your child may receive possible 
educational and learning benefits from his or her participation in this Study, and as a result of your child’s 
participation, educational products, in general, should benefit. 


4. Though Study results may be shared with you or your child throughout the Research Study, you waive 
any right to inspect and/or approve of the Work Product or how the Work Product may be used by Pearson. 


5. Should you agree to allow your child to participate in this Research Study, you acknowledge that your 
child will have access to Pearson Confidential Information. As such, you agree your child will not disclose 
any Pearson Confidential Information in any manner, in whole or in part, to any other party, or to use such 
Pearson Proprietary Information except for purposes of participating in the usability study. For purposes of 
this letter, “Pearson Confidential Information" means any information disclosed to your child by Pearson, 
either orally or in writing, relating to any products, services, or software code or functionalityowned, 
developed or being developed by or for Pearson or its affiliates, which specifically includes Pearson’s 
Equation Editor and any materials related to the Equation Editor including, but not limited to, source code, 
processes, designs, research, marketing methods or distribution plans, except to the extent that such 
information is already generally known or available to the general public or is disclosed by Pearson to third 
parties without restriction on disclosure. 


6. The following are considered Permitted Uses of Pearson’s Confidential Information by your child as a 
research participant: (i) to evaluate and/or test the Pearson Drawing Tool as instructed by Pearson during 
this Cog Lab Study, or (ii) for any other specific purpose that your child has been authorized to do by 
Pearson. Anything you’re your child does with Pearson’s Confidential Information outside the scope of the 
Permitted Uses will be considered a breach of this Agreement, unless your child has received prior written 
authorization from Pearson. 


7. At your direction or at your child’s request, your child may withdraw or terminate his or her 
participation in this Research Study at any time; however, Pearson will own all right, title and interest to 
any Work Product that has been created or submitted by your child at the time of your child’s withdrawal 
from the Study. 


8. You agree that any copyright or other intellectual property rights in the responses to the research 
activities described in this letter, including any suggestions, concepts or ideas that are provided to Pearson 
in the course of your child’s participation in this Research Study, will be owned by Pearson and are hereby 
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assigned to Pearson. You or your child will not be entitled to any compensation, royalties or attribution if 
Pearson utilizes any such responses, writings, suggestions, concepts or ideas in its products. You further 
agree that for copyright purposes, Pearson shall own all right, title, and interest in the audio, video, and/or 
screen recordings as a result of your child’s participation in this Research Study. 


9. As full compensation for your child’s participation and for the rights granted in this Release, Pearson 
agrees to provide your child with a $25.00 Walmart gift card at the end of his or her participation. 


If you have any questions regarding this release letter or your child’s participation in these activities, please call 
Rusty Brandt at 480-457-7429. If you agree to allow your child to participate in this Research Study pursuant to the 
terms set forth in this letter, please electronically sign and date below and submit the signed copy by the date of 
your child’s scheduled study session. Please make a copy of this letter for your records. 


We look forward to your child’s participation! 


Sincerely, 
Rusty Brandt 
Senior Researcher 


Pearson Design Research & Efficacy 


ACKNOWLEDGED AND AGREED: 


| am the parent or guardian of the person whose name appears below and who is younger than eighteen years old. | 
permit the participation of the person referred to below pursuant to the terms set forth in this letter. | also agree 
that the minor will be accompanied to the usability study by a responsible adult. These provisions shall carry the 
same force as if | had agreed to them on my own behalf. 


Mailing Address: 


Child’s Name (PRINT) 


Name of Parent/Guardian (PRINT) 


Signature of Parent/Guardian Date 
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Email Address Telephone Number 


APPENDIX C — VIDEO RECORDINGS 


Twelve of the thirteen participants have a video outlining their interaction with the PARCC 
drawing response interaction. 


All videos can be accessed in the folder Maryland Drawing Tools - Videos, found at 
https://pearson.box.com/s/h5xid2c4qjbnimpzuulm4blaqyoijdsy (Note: Videos are not yet 
accessible). 


APPENDIX D — SCREENSHOTS OF PARTICIPANT DRAWINGS 


Fine Motor Impairment 


Participant | Paper/Pencil Chromebook/iPad 


P1 
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Low Vision 
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We were not able to record this session. 


General Population 


Paper/Pencil iPad/Chromebook 
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Chromebook: 
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APPENDIX E — MODERATOR’S GUIDE 


November PARCC Drawing Tool CogLab: 
Moderator’s Guide 


Session Introduction 
1. Introductions — introduce team members: Rusty, Elizabeth 


2. Room Setup — we will be recording the session. We will capture video of the participant, 
the keyboard, and the computer screen. 


3. Purpose — today you are going to be looking at a tool to help students show their work 
and answer math questions. 


4. Feedback — we will be going through a series of activities. We want you to think out loud 
and feel comfortable sharing what you’re thinking. If you are feeling confused, or you have 
expectations that are not being met, we want to hear about that. Be as honest as possible. If 
you think something is awful, please say so. Don’t be shy, you won't hurt anyone’s feelings. 
We are very interested in understanding your ideas and opinions about what you are about 
to see, so please share freely. 


5. Our goal is to make well-designed products for everyone, so it’s important we 
understand what you think and what does and doesn’t work for you. 


6. Any Questions? 
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Overview 


Demographic Interview: Do you, or have you ever use a computer to complete math 
assignments? 


Scenario: Imagine that you’re about to take a test online. You will be expected to show 
all of your work within the test on the computer. 


Potential Other Questions: 


What type of alternative device are you using? 


Tasks 


Do participants successfully use the drawing response interaction to support their 
answer? Without moderator prompts? 

Do participants encounter any difficulties with the drawing response interaction 
features? With which features did the participant have the most difficulty? 

Do participants encounter any general technological difficulties with touch or 
mouse? Do they have difficulty manipulating the mouse and/or using the touch 
screen? 

Do participants intuitively know how to use the drawing response interaction 
features (pencil tool, straight line tool, line weight tool, color tool, eraser tools)? If 
not, which features did they not intuitively understand? 

What are the usability, access, or processing issues for special populations, in 
particular, students with fine motor impairments, low vision, and color blindness? 
Do participants use the different features/palettes of the tool, such as the 
undo/erase tools (if needed)? 

Do participants run out of room in the drawing response interaction? Do they 
erase to construct smaller graphics to fit the drawings in the box? 

Do participants locate the response area for drawing and text components? Is 
there a preference for placement? Is it problematic to have the locations move? 
Do users identify any other functionality they would like to see added to the 
drawing tool? 


General Questions: 
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Tell me what you're thinking. 

How would you use these tools to solve the problem? 

Is it working like you expect it to? 

Is there anything you find confusing or frustrating? 

Is there anything you'd like the tool to do that it doesn’t do now? 

What's frustrating you? What’s causing you to struggle? 

Is there anything missing, or would you like something to act/behave differently? 
What would you add to the drawing tool in order to help you solve this problem? 
What do you think of the location of the tool? 


Watch for 


Issues with clicking/touching buttons 
Issues with click, drag, release sequence 
Issues with scrolling 

Issues locating drawing box and/or textbox 
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Part 1: Basic drawing response interaction tasks (Tasks 1-3 


Task 1: Draw a Straight Line, Curved Line 
a. [After login] Look over this page and tell me what you think you can do here. 


1) How would you expect to be able to use this tool? 
2) What do you think each of the buttons are for? 

3) What would you expect to be able to do? 

4) Is there anything you find confusing? 


b. Go ahead and do what the question asks. 
1) What colors do you see in the palette (how many)? 
2) If the participant uses the straight line tool, was it faster? was it easier? 


Watch for 

Use of straight line vs pencil tool 

Number of colors participant can see in the palette 

Any difficulties using the drop-down menus or selecting buttons 
Any difficulties with the click, drag, release sequence 


O 


Now, complete the same task on paper. 
1) Was it easier or harder completing it on paper? 


Explain to participants: If you were doing it on paper, wouldn't be able to-- just do 
your best. Don’t worry about how they will score them-- precision (on paper, 
wouldn't have to be exact). 


Task 2: Draw a Circle 


a. Complete what the question asks. 
1) What do you think of your circle (are they satisfied with the accuracy)? 


b. Now, complete the same task on paper. 
1) Was it easier or harder completing it on paper? 


Task 3: Draw a Rectangle, Shade / Draw a Circle, Shade 
a. Complete what the question asks. 

1.) If the participant uses the straight line tool, was it faster? was it 
easier? Do you understand the difference between the pencil and straight 
line tool? 

2.) If participant does not use the straight line tool, have them try. What 
do you think about the pencil and straight line tool? Is one easier to use 
than the other? Why? 
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3.) If the participant uses the weight tool for shading, what did you think 
of the weight tool? 

4.) If the participant does not use the weight tool, ask them to use it. 
What did you think of that? 


Watch for 
e Use of straight line vs pencil tool 
e Use of weighted tool for shading 
e Difficulty with shading 


Part 2: PARCC-like items, Computer/iPad, Online Response (Tasks 4- 
9) 


Task 4: Picnic Tables 


a. Here’s an actual math problem you might encounter in a test. Go ahead and 
solve it as you would normally, if you were taking this test at school. 

1.) If the participant uses the straight line tool, was it faster? was it 
easier? Do you understand the difference between the pencil and straight 
line tool? 

2.) If participant does not use the straight line tool, have them try. What 
do you think about the pencil and straight line tool? Is one easier to use 
than the other? Why? 


Watch for 

e Use of straight line vs pencil tool (for picnic table drawing) 

e Use of weighted tool (for marking the people at the tables) 

e Difficulty with shading 

e If/how the participant uses the drawing box and textbox 

e VISUALLY IMPAIRED: If they find the equation editor box easily 
Task 5: Fractions 

a. Complete what the question asks 


Watch for 
e Whether or not the participant understands the question 
e I|f/how the participant uses the drawing box and textbox 
e VISUALLY IMPAIRED: If they find the equation editor box easily 


Task 6: Toy Cars 
a. Complete what the question asks 


Watch for 
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e Whether or not the participant understands the question 
e If/how the participant uses the drawing box and textbox 
e VISUALLY IMPAIRED: If they find the equation editor box easily 


Task 7: Areas of Tables 
a. Complete what the question asks 

1.) If the participant uses the straight line tool, was it faster? was it 
easier? Do you understand the difference between the pencil and straight 
line tool? 

2.) If participant does not use the straight line tool, have them try. What 
do you think about the pencil and straight line tool? Is one easier to use 
than the other? Why? 


Watch for 
e Whether or not the participant understands the question 
e Whether or not the participant uses the straight line tool 
e |f/how the participant uses the drawing box and textbox 
e VISUALLY IMPAIRED: If they find the equation editor box easily 


Task 8: Picture Array 
a. Complete what the question asks 
1.) If the participant struggles to draw the array, what’s frustrating you? 
What’s causing you to struggle? 
Watch for 
e Whether or not the participant understands the question 
e Whether or not the participant uses the weighted tool 
e Whether or not the participant runs out of room or has to erase and start over 
due to lack of room 
e  If/how the participant uses the drawing box and textbox 
e VISUALLY IMPAIRED: If they find the equation editor box easily 


Task 9: Beads 
a. Complete what the question asks 
1.) If the participant does not complete part B, are you done? Is there 
anything else you would do? 


Watch for 
e Whether or not the participant understands the question 
e |f/how the participant uses the drawing box and textbox 
e Whether or not the participant puts his/her answer in the Part B box 
e VISUALLY IMPAIRED: If they find the equation editor box easily 


Part 3: PARCC-like items, Computer/iPad, Paper Response (Tasks 
10-12) 
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e How do students solve problems on paper? 
e What are the main differences between solving on paper, and solving on 
computer/iPad? 


Tasks 10-12 
Say: For this next set of problems, we’re going to ask that you solve them using 
paper/pencil, as you would with a traditional test you might take at school. 


a. Complete what the question asks on paper 

1.) If the participant does not draw on the paper, can you create a drawing 
to illustrate what you are thinking? 

2.) Do you think you would draw the same thing if you were working on a 
computer instead of on paper? 

3.) Is it easier to draw on the computer/iPad or on paper? Which do you 
prefer? 

4.) What were you able to do on paper, that you weren't able to do on the 
computer/iPad? 


Watch for 
e Whether or not the participant understands the question 
e Whether or not the participant uses the paper/pencil to draw 


Post-Interview Questions 


[If not addressed in earlier conversations about drawing tool] 


a. Are there any online math tools, apps, etc. that you’ve used before that help you 
explain your work, or solve math problems? 


1.) If so, what are they? How do they help you?? 

2.) What can we do to make it easier for you to show your work/solve the 
problems? 

3.) [If not already addressed earlier] What would you add to the drawing tool 
in order to help you solve math problems? 


